Language identification of code Switching sentences and multilingual sentences of under-resourced languages by using multi structural word information

Language identification (LID) is a process to identify the languages used in a text or speech. Code switching is the switching of a language in a sentence or speech utterance. This paper focuses on LID of words in code switching sentences. Code switching can occur intersentential or intrasentential. The reasons why a writer switches from one language to another due to various reasons and among them are the inability to express opinion in a particular target language, to attract attention, to address different audience, habitual expressions and so on. The difficulty in identifying the languages of each word in a code switching sentence is because the languages have the same character set. In addition, code switching can happen in a sentence as short as a word or as long as a sentence. In this paper, we propose an automatic LID for words in code switching sentences by using multi structural word information (MUSWI) such as grapheme, syllable and word structure and calculate by using n-gram statistical model. The proposed MUSWI approach achieves 96.36% in term of accuracy on the code switching sentences, 99.07% on the multilingual sentences (non-code switching) which are under-resourced and closely related languages. Index Term: language identification, code switching, n-gram, multilingual, under-resourced languages, closely related languages